feat: Add cloud storage integration with S3 upload/download and snapshot management #107

tobidae-cb · 2025-08-06T04:59:13Z

🚀 Core Features Added:

Enhanced Snapshot Management System
- Optimized two-tier snapshot system for better performance
- Initial snapshots downloaded once and reused across tests
- Per-test snapshot copies using optimized rsync strategies
- Remote snapshot source support for cloud-based initial snapshots
Machine Information Collection
- Added machine metadata collection (type, provider, region, filesystem)
- Machine info included in benchmark results for better context
Report Backend API Improvements
- New Go-based REST API backend for serving benchmark data from S3
Frontend Enhancements
- Updated React frontend to work with new backend API
- Support for displaying machine information in reports
- Integration with cloud-stored benchmark results

🔧 Technical Improvements:

Optimized rsync strategies for faster snapshot copying
Better error handling and logging throughout the system
Structured configuration management for import/export operations
TTL-based caching for improved performance
Docker support for containerized deployment

cb-heimdall · 2025-08-06T04:59:18Z

✅ Heimdall Review Status

Requirement Status More Info

Reviews

✅ 1/1

Denominator calculation

Show calculation

1 if user is bot 0

1 if user is external 0

2 if repo is sensitive 0

From .codeflow.yml 1

Additional review requirements

Show calculation

Max	0

0

From CODEOWNERS 0

Global minimum 0

Max 1

1

1 if commit is unverified 0

Sum 1

…r run

meyer9

This looks awesome! I love all the new UI elements and how you integrated machine type/etc. Is there a way to make this agnostic to S3?

It looks like we're tying the report generation to S3 now, but users may want to use other cloud storage, or something like Github Actions.

export: downloads and uploads to s3 - can be done using aws s3 cp outside of the benchmark tool
serving from s3: this is just a static file server pulling from S3 - could just use an existing solution to serve static files

One philosophy I think we should adopt here is: do one thing and do it well. I think adding S3 download/upload/serving is nice to have, but I'm not sure it makes sense to include in the external repo that other teams may use who may not use S3.

The point of serving data from static files instead of a backend is that it makes deployment super simple. All of this work seems compatible with the static file interface, so adding S3 seems slightly unnecessary.

scripts/setup-initial-snapshot.sh

runner/clients/reth/client.go

runner/benchmark/snapshots.go

Pjrich1313

Have all POW to contract this work

meyer9

Left some more comments, but looks good overall!

I have a few suggestions:

remove AWS integration from this repo - this should probably just be in our internal repo
remove report backend from this repo - also can be in our internal repo - externally, people can use a static file system

Other than removing AWS dependency, I think the chain copy behavior is a little bit confusing. For ZFS, would I reuse_existing or chain_copy?

report/src/services/dataService.ts

benchmark/cmd/main.go

benchmark/config/import_config.go

report/backend/cmd/main.go

report/backend/internal/services/s3.go

report/src/components/RunList.tsx

runner/aws/s3.go

runner/benchmark/benchmark.go

runner/benchmark/matrix.go

meyer9

LG! Nothing blocking, just a few small suggestions - feel free to take or leave any of them.

meyer9 · 2025-10-02T20:24:07Z

benchmark/cmd/main.go

 			Description: "Runs benchmarks according to the specified config.",
 		},
 		{
 			Name:        "import-runs",


small nit: I think this file can be reverted

meyer9 · 2025-10-02T20:29:22Z

SNAPSHOT_SYSTEM.md

+
+## Configuration Format
+
+### New YAML Structure


I think we can remove this file. Maybe move some of this explanation into configs/examples/snapshot.yml

meyer9 · 2025-10-02T20:30:02Z

scripts/setup-base-snapshot.sh

@@ -0,0 +1,177 @@
+#!/bin/bash


maybe rename this to setup-base-snapshot since now it's specific to Base?

Approved review 3296275630 from meyer9 is now dismissed due to new commit. Re-request for approval.

feat: Add upload to aws and retrieving results from aws

b7a3663

tobidae-cb force-pushed the tobi/add-aws branch from e8ad7c4 to b7a3663 Compare August 6, 2025 05:26

feat: update how report backend works

715d03f

tobidae-cb force-pushed the tobi/add-aws branch from e11628b to 715d03f Compare August 12, 2025 20:27

tobidae-cb added 4 commits August 15, 2025 14:45

Setup initial snapshot from remote source and copy per tests

77038a7

Add frontend build

6dfc2b1

Merge remote-tracking branch 'origin' into tobi/add-aws

836ecd1

Add machine info

17c4a1d

tobidae-cb changed the title ~~feat: Add download/upload capabilities with cloud storage~~ feat: Add cloud storage integration with S3 upload/download and snapshot management Sep 12, 2025

tobidae-cb marked this pull request as ready for review September 12, 2025 18:57

More frontend changes

c758c68

tobidae-cb force-pushed the tobi/add-aws branch from 92cab4c to c758c68 Compare September 12, 2025 18:58

tobidae-cb added 5 commits September 16, 2025 11:00

Use debug setHead for quick snapshot rollbacks, add client version pe…

2d7ec86

…r run

Remove redundant client version

9cbf7c2

Add additional snapshot method

67b4cd7

Fix warnings

d0f2321

Fix lint issues

6b94a74

tobidae-cb requested a review from meyer9 September 19, 2025 16:22

meyer9 reviewed Sep 19, 2025

View reviewed changes

scripts/setup-initial-snapshot.sh Outdated Show resolved Hide resolved

runner/clients/reth/client.go Show resolved Hide resolved

runner/benchmark/snapshots.go Show resolved Hide resolved

Pjrich1313 reviewed Sep 26, 2025

View reviewed changes

tobidae-cb added 3 commits September 26, 2025 10:02

Update frontend

6a0d4aa

Update output dir

9457e53

Merge remote-tracking branch 'origin' into tobi/add-aws

2d7ae0c

meyer9 reviewed Sep 29, 2025

View reviewed changes

tobidae-cb force-pushed the tobi/add-aws branch 2 times, most recently from 138892f to a3364ad Compare September 30, 2025 17:43

Remove backend and ref to s3, fix lint

a6db049

tobidae-cb force-pushed the tobi/add-aws branch from bb16e8d to a6db049 Compare October 2, 2025 19:17

Remove all refs to s3

86dafdd

meyer9 previously approved these changes Oct 2, 2025

View reviewed changes

revert main.go, rename and delete some files

da990e5

More changes

addb893

meyer9 approved these changes Oct 2, 2025

View reviewed changes

tobidae-cb enabled auto-merge (squash) October 2, 2025 20:41

tobidae-cb merged commit 5a8621a into main Oct 2, 2025
10 checks passed

tobidae-cb deleted the tobi/add-aws branch October 2, 2025 20:56

feat: Add cloud storage integration with S3 upload/download and snapshot management #107

feat: Add cloud storage integration with S3 upload/download and snapshot management #107

Uh oh!

Conversation

tobidae-cb commented Aug 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🚀 Core Features Added:

🔧 Technical Improvements:

Uh oh!

cb-heimdall commented Aug 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Heimdall Review Status

Uh oh!

meyer9 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Pjrich1313 left a comment

Choose a reason for hiding this comment

Uh oh!

meyer9 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

meyer9 left a comment

Choose a reason for hiding this comment

Uh oh!

meyer9 Oct 2, 2025

Choose a reason for hiding this comment

Uh oh!

meyer9 Oct 2, 2025

Choose a reason for hiding this comment

Uh oh!

meyer9 Oct 2, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

tobidae-cb commented Aug 6, 2025 •

edited

Loading

cb-heimdall commented Aug 6, 2025 •

edited

Loading